1. Field of the Invention
The present invention relates to an apparatus and a method for performing streaming distribution of digitized multimedia data such as audio, video, etc.
2. Description of the Related Art
The MPEG-4 file format (Moving Picture Experts Group phase 4 File Format (MP4)) is the international standard file format specification defined by the ISO/IEC (International Organization for Standardization/International Electrotechnical Commission) 14496-1 (MPEG-4 Systems Part), and the specification for storing audio/video codec data which is an international standard of the ISO/IEC.
When a server uses an MP4, file as a container of streaming data, hint information is used as auxiliary information for easy generation of a transport packet by the server when streaming distribution is performed. MP4 is designed to store also the hint information.
The specification of the hint information is currently being prepared by the ISO/IEC 14496-1, and the operation is being performed in accordance with the specifications of the QuickTime (registered trademark) of Apple Inc. The specifications of the MP4 have also been improved and defined based on the specifications of the QuickTime (registered trademark).
In the hint information, a data packetizing method (dividing method) is mainly described, and the server packetizes video codec data based on the described method. In the data format of the hint information, average bit rate information and maximum bit rate information for use when video codec data is treated in the packetizing process and the streaming distribution can be described.
The server can select hint information according to the bit rate information based on the available band of a distribution network and the performance of a receiving terminal. Thus, a scalable streaming distribution can be realized.
FIG. 1 shows the configuration of the above-mentioned conventional streaming server. A server 21 shown in FIG. 1 includes a storage device 31, a packet generation unit 32, a transmission unit 33, and a hint information selector 34.
The storage device 31 is, for example, a hard disk drive, and stores a plurality of MP4 files 35. Each MP4 file 35 contains codec data 36 and plural pieces of hint information 37. The hint information selector 34 selects hint information based on the bit rate information described in the hint information 37 and the indicated distributing method, and the packet generation unit 32 packetizes the codec data 36 according to the selected hint information. Then, the transmission unit 33 transmits a generated packet to a receiving terminal through a communications network 22.
Thus, in the above-mentioned streaming distributing method, it is not necessary to divide in advance the video codec data in various packet formats and store the video codec data on a disk for each packet format. Instead, it is only necessary to store an amount of video codec data and hint information about various packetizing methods, thereby indicating an effect is improving the disk storage efficiency.
FIG. 2 shows a method for generating the MP4 file 35. A MP4 generator 44 generates the MP4 file 35 from video codec data 41 and audio codec data 42, and a hint information generator 45 generates hint information according to hint information generation designation information 43.
The MP4 file 35 generally includes moov 47 for managing various index information and mdat 48 for managing the entity of codec data. The moov 47 includes a plurality of track management units (trak) corresponding to the respective entities of codec data of the mdat 48.
For example, media track 49 manages the index of video codec data 53, and a media track 50 manages the index of audio codec data 54.
Each media track manages identification information (ID) 61, referenced identification information (Ref ID) 62, and index information (idx information) 63 as a list as shown in FIG. 3. The index information 63 is provided for each piece of codec data (access unit) of the smallest unit required in regeneration, and the n-th index information (idx information #n) includes the position information (offset Mn, size Mn) and the time information (duration Mn) in the file of the n-th access unit (AU#n).
When the hint information is stored, it is divided into a hint track in the moov 47 and a hint sample in the mdat 48. The hint sample is the information about the packetizing method for each access unit of the codec data. In FIG. 2, a hint track 51 manages a hint sample 55 for the video codec data 53 and a hint track 52 manages a hint sample 56 for the audio codec data 54.
Each hint track manages identification information (ID) 71, reference identification information (RefID) 72, bit rate information 73, and index information (idx information) 74 as a list as shown in FIG. 3. The reference identification information 72 is the identification information about a media track referenced by a hint track, and the bit rate information 73 is bit rate information about the streaming distribution in the packetizing process.
The index information 74 is provided for each packetizing method of the access unit managed by a referenced media track, and the m-th index information (idx information #m) includes the position information (offset Mm, size Mm) and the time information (duration Mm) about the m-th packetizing method (sample#m).
The sample#m indicates the dividing method of an access unit, and manages reference identification information (RefID) 75, data size (length) 76, data position (offset) 77, and identification information (AU#) 78 for each piece of the divided data.
The packet generation unit 32 shown in FIG. 1 performs a packetizing process on each access unit of the codec data managed by the referenced media track of the hint track using the information about a hint sample managed by the hint track as shown in FIG. 4. Thus, a packet including a transport header 81 and a payload 82 is generated.
Then, it is assumed that layered codec data and its hint information are stored in the above-mentioned MP4 file. The layered codec data process has no problem of extreme image degradation, but is a codec system capable of easily and stepwise adding or thinning data (stepwise image quality change) depending on the network communications status, and can be MPEG-1, MPEG-2, and MPEG-4. In the MPEG-4, an FGS (fine granular scalability) profile is well known.
As for the layer of the finest granularity of the layered codec data, a plurality of variations of hint information can be assumed depending on the variation of the layer-dividing method for determining how many layers are to be set for the thinning operation, and how each layer is paired. The variation of hint information increases depending on which is used, unicast distribution or multicast distribution.
FIG. 5 shows a data flow when unicast distribution is performed on the virtual layered video codec data. In this case, layered video codec data 91 includes three layers L0, L1, and L2, and data is distributed along one channel from the server 21 to each client.
For example, the server 21 generates a packet 101 according to the hint information #0 from the layered video codec data 91, and distributes it to a client 93 through a router 92. Similarly, packets 102 and 103 are generated according to the hint information #1 and #2, and distributes them to clients 94 and 95.
In this case, the image quality is associated with the bit rate. The client 93 receives video of the lowest image quality at the lowest bit rate. The client 95 receives video of the highest image quality at the highest bit rate. Therefore, the server 21 can select the hint information depending on the image quality by referring to the bit rate information contained in each piece of hint information.
On the other hand, when the multicast distribution is performed, it is assumed that the data flow as shown in FIG. 6 occurs. In this case, the data of each layer is distributed from the server 21 to each client along a different channel.
For example, each server 21 generates a packet 104 according to the hint information #0 (common with the hint information for the packet 101 shown in FIG. 5) from the layered video codec data 91, and transmits it to a multicast router 96. Similarly, packets 105 and 106 are generated according to the hint information #3 and #4 respectively from the layered video codec data 91, and transmitted to the multicast router 96.
The multicast router 96 distributes the packet 104 to the client 93, distributes the packets 104 and 105 to the client 94, and distributes the packets 104, 105, and 106 to the client 95. Therefore, the client 93 receives the video of the lowest image quality, and the client 95 receives the video of the highest image quality.
However, unlike the unicast distribution, since the bit rate does not reflect the image quality, the server 21 cannot select the appropriate hint information only by referring to the bit rate information.
FIG. 7 shows the configuration of a virtual MP4 file for distribution of the layered video codec data as shown in FIGS. 5 and 6. The moov 47 stores media track 111 for management of layered video codec data 117 and the hint tracks 112 through 116 for management of hint samples 118 through 122. The hint samples 118, 119, 120, 121, and 122 hold the packetizing methods of the hint information #0, #1, #2, #3, and #4 respectively.
Thus, plural pieces of hint information can be stored according to the specifications of the current MP4. However, there is a problem as follows.
(1) Although plural pieces of hint information in a layer-dividing method are stored for the layered video codec data managed by the media track, how the layered video codec data is layer-divided according to each hint information cannot be indicated. Therefore, the hint information selector 34 cannot select the hint information matching the service, and the stream data generated according to the hint information cannot be associated with the distribution port matching the service.
(2) Even if plural groups of hint information corresponding to a plurality of layer-dividing methods are stored using the above-mentioned plural pieces of hint information as a group, plural pieces of hint information cannot be managed as a group. Therefore, the hint information selector 34 cannot select a group of hint information matching the service.
(3) Since there is no information indicating a group for use in the unicast distribution or a group for use in the multicast distribution relating to the problem (2) above, the hint information selector 34 cannot select a group of hint information matching the distributing method.